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Abstract 

Background: The objective of this study was to develop a rating tool for policy makers to prioritize breast cancer 
interventions in low- and middle- income countries (LMICs), based on a simple multi-criteria decision analysis 
(MCDA) approach. The definition and identification of criteria play a key role in MCDA, and our rating tool could be 
used as part of a broader priority setting exercise in a local setting. This tool may contribute to a more transparent 
priority-setting process and fairer decision-making in future breast cancer policy development. 

Methods: First, an expert panel (n = 5) discussed key considerations for tool development. A literature review 
followed to inventory all relevant criteria and construct an initial set of criteria. A Delphi study was then performed 
and questionnaires used to discuss a final list of criteria with clear definitions and potential scoring scales. For this 
Delphi study, multiple breast cancer policy and priority-setting experts from different LMICs were selected and invited 
by the World Health Organization. Fifteen international experts participated in all three Delphi rounds to assess and 
evaluate each criterion. 

Results: This study resulted in a preliminary rating tool for assessing breast cancer interventions in LMICs. The tool 
consists of 10 carefully crafted criteria (effectiveness, quality of the evidence, magnitude of individual health impact, 
acceptability, cost-effectiveness, technical complexity, affordability, safety, geographical coverage, and accessibility), with 
clear definitions and potential scoring scales. 

Conclusions: This study describes the development of a rating tool to assess breast cancer interventions in LMICs. Our 
tool can offer supporting knowledge for the use or development of rating tools as part of a broader (MCDA based) 
priority setting exercise in local settings. Further steps for improving the tool are proposed and should lead to its useful 
adoption in LMICs. 

Keywords: Multi-criteria decision analysis, Priority setting, Breast cancer 



Background 

As the second most common cancer in the world and 
the most common cancer in women, breast cancer is an 
important health problem globally [1]. Although it was 
originally considered to be a disease of the developed 
world, low- and middle-income countries (LMICs) are 
experiencing large increases in incidence [2]. Mortality- 
to-incidence rates remain relatively high in these areas 
[3], possibly due to relatively poor breast cancer control 
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strategies (e.g. awareness raising, early detection, treat- 
ment) and differences in cultural beliefs [2,4], Because 
strong early detection programs are beneficial, the 
World Health Organization (WHO) seeks to improve 
appropriate breast cancer control programs in LMICs. 

Cost-effectiveness analyses (CEAs), based on the maxi- 
mization of health benefits, have often been used for the 
selection of breast cancer control strategies. To provide 
an evidence base for the cost-effectiveness of breast can- 
cer interventions in LMICs, a consortium of the WHO, 
Erasmus University Rotterdam, the Radboud University 
Nijmegen Medical Center, and the Susan G. Komen for 
the Cure Foundation initiated an international study in 
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2010 [5]. Such CEAs may help governments decide how 
to spend scarce health care resources more efficiently. 
However, decision-makers often deviate from CEA re- 
sults because other principles such as equal treatment 
and priority to the worst-off [6-8] and other factors like 
feasibility or acceptability influence decisions, as well 
[9-11]. Ignorance about these criteria may induce imple- 
mentation problems or inequality among certain patient 
groups [12-14]. 

Multi Criteria Decision Analysis (MCDA) is a well- 
accepted framework that can simultaneously assess 
multiple criteria for priority setting of interventions 
[15]. Different approaches of MCDA are proposed but 
contain at least the following elements: 1) selection of 
relevant interventions, 2) selection of criteria for priority 
setting, 3) collecting evidence and rating the performance 
of interventions on selected criteria, 4) deliberation on the 
evidence and performance of interventions with the aim to 
select the best interventions for implementation [16-19]. 

Several studies have shown the potential of MCDA in 
prioritizing health interventions, however, it has not yet 
been applied for the selection of breast cancer control 
interventions [20-23]. Recently, MCDA has been criticized 
for being technocratic and conceptually challenging for 
local decision makers [24]. Therefore, the development of 
a tool to support local policy makers in selecting criteria 
and in rating the performance of interventions on these 
criteria is warranted. 

The objective of this study is to develop a rating tool 
to assess breast cancer interventions along the con- 
tinuum of care, within the context of the overarching 
breast cancer CEA project [5]. The rating tool will be 
composed of criteria, criteria definitions, criteria weights 
and rating scales to measure the overall impact of breast 
cancer interventions and support the priority setting 
process. Such a rating tool can be used in a broader, 
MCDA based, priority setting process to develop cancer 
control strategies in a local setting. 

Methods 

To develop the rating tool we established an expert 
panel (n = 5) of breast cancer and priority-setting experts 
from WHO and the Radboud University Nijmegen 
Medical Centre. The expert panel consisted of two health 
economists, a scientific researcher, a public health special- 
ist and a student on health technology assessment. Three 
of the experts are co-authors of this article (KV, SZ and 
JL). A detailed overview of the considerations made by the 
expert panel in the development of the rating tool is pro- 
vided as additional information (Additional file 1). Below 
we describe the most important steps that were taken to 
develop the tool. 

A literature search using PubMed and Google Scholar 
was performed for the identification of a first set of 



predefined criteria. Different combinations of the terms 
criteria , Values', 'factors', priority setting', 'decision making', 
and policy making' were used as the query. The expert 
panel discussed the list in order to avoid overlap among 
the criteria. For the remaining criteria, clear definitions 
were defined with the help of glossaries and documents 
published by the WHO [25-27]. 

To develop the scoring scales, another literature study 
was performed for each criterion of this predefined list. 
When no or little information was available, scoring 
scales were mainly based on discussions with the expert 
panel. 

The Delphi study 

The list of predefined criteria and scoring scales was fur- 
ther refined by the opinion of experts from all over the 
world. A Delphi study was chosen because of the anonym- 
ity of participants, the opportunity to include participants 
globally, and the time and money available to conduct the 
study [28]. Delphi studies have proven to be appropriate 
for finding a core list of evaluation criteria [29]. 

Participants 

Experts were selected following WHO selection criteria 
that include a balanced geographical and gender repre- 
sentation, expertise in the technical area (particularly in 
LMICs), and absence of any relevant interest in the per- 
sonal declaration of interest form. Twenty-nine experts 
with expertise in priority setting or breast cancer pol- 
icies in LMICs were involved, ensuring methodological 
as well as substantive quality. Experts were identified by 
approaching authors of relevant articles and by snowball 
sampling. Among the experts there were epidemiologists, 
cancer survivors, pathologists, guideline-developers, public 
health specialists, radiotherapists, surgeons, researchers, 
managers, strategists and ethicists. 

First round 

In this round, the list with criteria based on the performed 
literature study was presented to the participants. The 
participants were asked to score the criteria on five-point 
Likert scales, according to whether they agreed that in- 
terventions scoring high on the criterion should be more 
prioritized (1 = strongly disagree, 2 = disagree, 3 = neutral, 
4 = agree, 5 = strongly agree). The experts could give 
comments on the list and mention whether important 
criteria were missing. In addition, the definitions and 
scoring scales of the criteria were presented and partici- 
pants were asked to provide comments. Likert scales were 
chosen for this first round because this method is reviewed 
as acceptable for a Delphi study and is simple and easy to 
perform [30]. 
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Second round 

This second round showed the scores and comments 
given in the first round, together with the adaptations to 
which they had led. Subsequently, participants were asked 
whether they agreed on the adaptations and if they could 
clarify their answers. 

Third round 

Based on the proportion of participants agreeing on the ad- 
aptations made after the first round and on the comments 
provided, some final changes were made to the criteria list. 
These final criteria and their definitions and scoring scales 
were shown to the participants, who were asked whether 
they agreed that this final list contained the most relevant 
criteria for the prioritization of breast cancer interventions. 
Furthermore, participants were asked to divide 100 points 
over the criteria according to their relative importance for 
the evaluation of breast cancer interventions. 

The analysis 

The analysis of the answers was both quantitative and 
qualitative. After the first Delphi round, mean and median 
scores on the Likert scales for "the importance" of criteria 
were calculated. The second round resulted in a percentage 
of participants who agreed with the suggested adaptations. 
After the third round, the mean and median weights given 
according to the importance of criteria were calculated. All 
participant comments were quantitatively analyzed. 

Results 

The literature search on criteria resulted in a total of 33 
criteria (Figure 1). After the expert panel discussed these 



criteria, nine criteria remained for the Delphi study. 
Two criteria, effectiveness and feasibility, were divided 
into three and four subcomponents. For each of these 
nine criteria and the subcomponents, a definition and a 
potential scoring scale were defined. 

Participants 

Out of 72 experts who were asked, 29 were willing to 
participate. Of these, 17 were experts on priority setting, 
and 12 were experts on breast cancer policies. The first 
questionnaire was completed by 23 participants, the 
second questionnaire by 19 participants, and the third 
questionnaire by 15 participants. Reasons for not com- 
pleting a questionnaire were private circumstances and 
disagreement with the aim of this research (n = 1). Most 
participants, however, gave no reason. 

First round 

Based on the results of the first round, two criteria 
('Severity of breast cancer' and 'Age') and one of the 
components ('The time until the effect emerges') were 
suggested for removal; two of the components were 
suggested to be combined with two criteria; and all the 
other components were suggested for separation into 
different criteria. Also, a new criterion was suggested 
('Political will'), two definitions were refined, and four 
scoring scales were adapted. These adaptations led to a 
list of 10 criteria. For all criteria, except for the criter- 
ion 'Effectiveness', there was divergence in Likert scale 
scores. The average and median Likert scale values and 
most important comments on the criteria are shown in 
Table 1. 



Stage 

Literature study 



Expert panel 



Delphi first round 



)elphi 



Delphi second round 



0 



9 criteria, one with 
3 and one with 4 
subcomponents 



Results Number of criteria 

'Effectiveness', 'Time until effect emerges', 'Quality of evidence', 'Efficiency', 'Acceptability', 33 criteria 
'Appropriateness', 'Accessibility', 'Competence', 'Continuity', 'Safety', 'Affordability', 'Impact on 
health budget', 'Human resources', 'Sustained access to materials and drugs', 'Magnitude of 
individual benefit', Type of medical service', 'Severity of disease-status', 'Capacity to benefit', 
'Lifetime equality of health', 'Age of target group', 'Life threatening disease-status', 'Rare 
disease-status'/ 'need of orphan drugs', 'Responsibility for health status', 'SES (wealth, income 
and education) of target group', 'Area of residence', 'Race'/ 'Ethnicity and religion of target 
group', 'Catastrophic levels of health expenditure', 'Productivity losses', 'Positive and negative 
externalities', 'Political willingness', 'Funding sources'/ 'Donors', 'Media attention', 'International 
priorities' 

'Effectiveness' ('Size of the effectiveness', 'Certainty of the evidence' and 'Time until the effect 
emerges'), 'Cost-effectiveness', 'Feasibility' ('Reach', Technical complexity', 'Capital intensity' 
and 'Cultural acceptability'), 'Safety', 'Accessibility', 'Severity of breast cancer', 'Age', 
'Magnitude of individual health impact', 'Catastrophic health expenditures' 

- 'Severity of breast cancer' and 'Age' removed 

- 'Time until the effect emerges' removed 

- 'Size of the effectiveness' and 'Certainty of the evidence' separated as individual criteria 

- 'Cultural acceptability' combined with 'Safety' and 'Reach' combined with 'Accessibility' 

- 'Technical complexity' and 'Capital intensity' separated as individual criteria 

- 'Political will' included as a new criteria 

- 'The certainty of the evidence' renamed into The strength of the evidence', 'Accessibility' 
renamed into 'Equal access', 'Capital intensity' renamed into 'Affordability', 'Cultural acceptability' 
renamed into 'Acceptability' 

- 'Political will' removed 

- 'Equal access' divided into two different criteria: 'Accessibility' and 'Geographical coverage' 

- 'Safety' divided into two different criteria: 'Safety' and 'Acceptability' 

- 'Catastrophic health expenditures' combined with 'Accessibility' 

- 'The strength of the evidence' renamed into 'The quality of the evidence' 

'Effectiveness', 'Quality of evidence', 'Magnitude of individual health impact', 'Acceptability', 
'Cost-effectiveness', 'Technical complexity', 'Affordability', 'Safety', 'Geographical coverage', 
'Accessibility' 



Figure 1 Overview of the development of the criteria list. 
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Second round 

Based on the results of the second round, the new sug- 
gested criterion ('Political will') was removed again be- 
cause participants argued that political will would also 
depend on the results of interventions on the other cri- 
teria; political will changes too often; and MCDA aims 
at a more fair priority-setting process while political 



will might even be clearly unfair. Two criteria ('Equal 
access' and Acceptability) were separated into two dif- 
ferent criteria ('Geographical coverage' and 'Accessibility'; 
Acceptability' and 'Safety'); two criteria were combined 
('Catastrophic health expenditures' and Acceptability'); 
and some small refinements to most of the definitions and 
scoring scales were made. An overview of the changes 



Table 1 Initial criteria including Likert scores and important comments given in the Delphi study 




Average 

Mlcprt croupe 


Median 

likpft crnrpc 

l/ACl 1 JLU/Cj 


Range of 

likert crnrpc 
f f rvci i jlu/ cj 


Most important comments 


Effectiveness 


4.75 


5 


4-5 


Effectiveness is covered by its components. Effectiveness should 
therefore be removed and its components should be independent 
criteria, otherwise they will overlap. 


Size of effectiveness 


4.70 


5 


3-5 


No important comments. 


Certainty of the 
evidence 


4.35 


5 


1-5 


Not related to effectiveness only. The strength of the evidence varies 
by criterion for any given intervention. Much simpler and effective to 
include considerations of certainty of evidence in assigning scores for 
all given criterion. 


Time until the effect 
emerges 


3.09 


3 


1-5 


Time preference for immediate effects goes against principles of 
intergenerational equity, and is especially inappropriate for preventive 
services. Therefore this criterion should be removed. 


Cost-effectiveness 


4.25 


4.5 


1-5 


MCDA might replace C/E. We can have costs but "effectiveness" is 
defined by the sum of the criteria so adding this criterion introduces 
double-counting. 

Efficiency cannot be replaced by costs since higher costs do not per 
se mean lower efficiency as the effectiveness may be higher. 


Feasibility 


4.23 


4 


2-5 


This should be four different criteria, otherwise they will overlap each 
other. 


Reach 


4.46 


5 


2-5 


See comments accessibility. 


Technical complexity 


3.5 


3.5 


1-5 


No important comments. 


Capital intensity 


3.75 


4 


1-5 


This criterion should not be limited to capital costs but also explicitly 
include operating costs required from the health system. 


C\ ilti irsl srrpntahilitv 

v_uiiuiai a^-^-c: vj la vj\ i i i y 


4.13 


4.5 


1-5 


Nn imnnrtsnt rommpn"K 

INU II 1 IkJ^JI LUI 11 LUI 1 II 1 l<3l llJ. 


Safety 


4 


4 


2-5 


The importance of safety may vary with respect to whose safety 
(provider vs. patient) and what is at stake, while the level of 
acceptability may remain the same. Therefore acceptability and safety 
should be kept separated. 


Accessibility 


4.33 


4.5 


1-5 


Accessibility due to geographical coverage of an intervention ('Reach') 
is not the same as accessibility due to socio-economic status. Therefore 
this criterion should be about equal access for patients with different 
socio-economic status, while geographical coverage should be covered 
by another criterion ('Reach') 


Severity of breast cancer 


3.26 


3 


1-5 


Of course 1 think that palliative care is very important. On the other hand, 
if you do nothing for all the people with earlier stage cancer, the cancer 
will progress and they will all need palliative care. So you could treat 
people with stage 1 or 2 cancer and most of them will not experience 
late stage cancer, therefore will not need palliative care. I guess I don't 
find this a useful way to think about breast cancer. 


Age 


3.29 


3.5 


1-5 


Ages of patients with breast cancer don't seem appropriate even if one 
wanted to create prioritized age groups, which I wouldn't. 


Magnitude of individual 
health impact 


3.83 


4 


1-5 


No important comments. 


Catastrophic health 
expenditures 


4.17 


5 


1-5 


Affordability is about whether the health system can afford an intervention 
and catastrophic health expenditures is about whether patients can afford 
it. Extreme health expenditures might however be covered by accessibility, 
because patients with lower socio-economic status cannot afford high 
health expenditures. 
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made in the criteria list is shown in Figure 1. The second 
round resulted in a final list of 10 criteria (Table 2). 

Third round 

All participants agreed that the list after the second round 
covered the most relevant criteria for the prioritization of 
breast cancer interventions. Three participants mentioned, 
however, that some criteria might be still overlapping. As 
one participant noted: "Doing the relative weighting exer- 
cise above, I realized that some criteria are overlapping 
and it was difficult to assess independent relative weights 
to them; for example, effectiveness and quality of the evi- 
dence' are inseparable whereas we would not perhaps say 
something is effective if the quality of the evidence is 
weak". There were also doubts about cost-effectiveness' 
being covered by the affordability' and effectiveness' and 
about safety' being covered by effectiveness' and 'geo- 
graphical coverage' being covered by effectiveness'. The 
criterion 'geographical coverage' was rated relatively low in 
its importance for the evaluation of breast cancer inter- 
ventions, followed by safety' and affordability', respect- 
ively. The importance of the criterion 'Effectiveness' was 
rated highest (Table 2). 

Discussion 

This study describes the development of a rating tool to 
measure the impact of breast cancer interventions based 
on multiple criteria in LMICs. Ten criteria, including defi- 
nitions and potential scoring scales, have been indicated. 
The results of this study show that effectiveness, quality 
of the evidence, magnitude of individual health impact, 
acceptability, cost-effectiveness, technical complexity, af- 
fordability, safety, geographical coverage, and accessibility 
seem to be important principles in the selection of breast 
cancer control strategies. Although selecting and defining 
interventions and criteria for breast cancer control is con- 
text specific, we think that this rating tool can be a starting 
point for local policy makers as part of a broader, MCDA 
based, priority setting process. 

Use of the tool in a LMIC 

The tool could be used as part of the integrated MCDA 
and accountability for reasonableness (A4R) approach 
for priority setting, recently proposed by Baltussen et al. 
[16] (Figure 2). This new approach combines strong 
components of both frameworks and requires a set-up 
of a multi-stakeholder consultation panel (step one). In 
this way a democratic learning process is started in 
which stakeholders are involved in all steps of the priority 
setting. Compared to the stand-alone MCDA framework, 
this combined approach may increase the acceptance of 
decisions among stakeholders and the likelihood of imple- 
mentation of prioritized programs. The rating tool can be 
part of step two and three (Figure 1) of the approach 



that aim to identify criteria for priority setting and 
assess (i.e. rate) the performance of interventions on 
the selected criteria. 

An important next step in the local use of the rating 
tool is to investigate how the tool and its components 
are understood in LMICs in a pilot study. Users of the 
tool could for example select relevant stakeholders (e.g. 
patients, lay-people policy makers, caregivers, public 
health specialists) and establish a consultation panel 
(step 1). These stakeholders could discuss the interven- 
tions, criteria, the attitude of decision-makers against 
the criteria and scoring scales using democratic elements 
(e.g. Nominal Group Technique) (step 2). After the col- 
lection of all relevant (local) evidence, the users could 
use our tool as an input for a performance matrix (step 3), 
and then interpret and deliberate on the results of this 
matrix (step 4 and 5). Users should be well informed and 
plan enough time for this process, and should try to en- 
sure that the tool is perceived as a simple and legitimate 
way to frame policy discussions in a more rapid and 
balanced manner. The potential of this tool could also 
be investigated for other cancers. 

Limitations of the study 

Our study has a number of limitations. First, prior to the 
Delphi study, the expert panel made a selection of 9 
criteria out of 33 criteria. This selection was based on 
overlap between criteria and whether criteria would be 
appropriate for the selection of breast cancer interven- 
tions. However, there is no certainty that personal pref- 
erences did not lead to bias in this selection. 

Second, we used Delphi studies to define a list with 
core criteria including definitions and scoring scales. 
The Delphi method ensures participant anonymity and 
provides enough time to properly consider one's own 
answers and those of others. However, the Delphi ques- 
tionnaire may not allow for adequate elaboration on 
difficult concepts such as equity and social welfare. 
Also, Delphi questionnaires can be relatively time con- 
suming, which may have partly caused 14 participants 
to withdraw from this study. We do not expect that 
these withdrawals biased the results because they varied 
in gender and type of expert and the number of com- 
ments remained high in each questionnaire. 

Third, the wide variety of comments and views of the 
participants made us aware of the difficulties in developing 
a clear, consensus-based, non-overlapping criteria list and 
scoring scales. There are many possible compositions and 
definitions of criteria [31-33]. Besides there are many ways 
to divide a scoring scale into different categories and this 
also depends on the variability of the interventions that 
are considered (i.e. discriminative power of the scoring 
scale). Further research should focus on more informed 
contextualized categories for scoring scales. 



Table 2 Final criteria list for the prioritization of breast cancer interventions including weights 



Definition 



Potential scoring scales 



Average weight* 
(min-max) 



Median 
weight 



Effectiveness [31-35] 



Quality of the evidence 

[31,32,35,37,38] 



Effectiveness is the extent to which an intervention impacts the 
most relevant health-related outcomes (e.g. time to recurrence 
or healthy life years gained). In comparison of effectiveness of 
interventions, it is important to note that the most relevant 
health-related outcome should be consistent for all interventions 
under consideration [25]. 

The risk of bias and the extent of the confidence that the 
evidence is adequate to support a particular decision or 
recommendation [39]. 



Magnitude of individual 
health impact [32,38] 



Interventions offering small benefits for many may be viewed 
differently from those offering large benefits for a few. When 
one of the two is preferred above the other, interventions 
providing the preferred effect (concentrated or dispersed) might 
be more prioritized [32]. 



Size of the effect (e.g. in a population of 1 million people): 17.33 (5-50) 15 

0 less effective (e.g. < 50 healthy life years gained a year) 

1 effective (e.g. > 50 < 100 healthy life years gained a year) 

2 very effective (e.g. > 100 healthy life years gained a year) [36] 

0 very little or limited confidence in the evidence: the estimated 1 1.93 (0-20) 12 
values may be substantially different from the outcomes in reality 

1 moderately confident about the evidence: The estimated values 
are likely to be close to the outcomes in reality, but there is a 
possibility that it is different 

2 very confident that the estimated values lie close to the 
outcomes in reality [39] 

Scoring scale a could be used in the case that local stakeholders 8.60 (0-25) 10 
decide that large individual health benefits are preferred above 
helping more people. 

Scoring scale b could be used in the case that local stakeholders 
decide that helping more people is preferred above large 
individual health benefits. 



Acceptability 

[26,34,35,38] 



Cost-effectiveness 

[31,32,35,37,40] 



The extent to which the intervention is judged as suitable, 
satisfying or attractive by different stakeholder groups 
(e.g. patients, providers or politicians). The acceptability 
depends on people their norms, beliefs and values [26,40]. 



The capacity to produce the maximum output for a given 
monetary input [25]. 



0 small individual health impact 

1 moderate individual health impact 

2 large individual health impact 
b 

0 benefits for just a few people. 

1 benefits for a moderate number of people 

2 benefits for many people 

0 the intervention is not accepted by some people and it is not 
likely that this can be changed 

1 the intervention is not accepted by some people but it is likely 
that this can be changed with some extra effort (e.g. special 
education) 

2 the intervention is accepted by almost all people 

0 not cost-effective (e.g. costs per gained healthy life year are 
above 3*Gross Domestic Product (GDP) per capita) 

1 cost-effective (e.g. costs per gained healthy life year are below 
3*GDP per capita) 

2 highly cost-effective (e.g. costs per gained healthy life year are 
below 1*GDP per capita) [41] 



8.67 (5-15) 



10 



12.4 (0-25) 



15 



Table 2 Final criteria list for the prioritization of breast cancer interventions including weights (Continued) 



Technical complexity 

[26,34] 



Affordability 

[26,31,34,35,38] 



Safety [31,34] 



Geographical coverage 

[26,32,34,35] 



Accessibility [32,37] 



Other types of inputs required in addition to monetary 
nputs to implement and to keep providing the intervention. 
(These include human resource requirements, both 
quantitative and qualitative, and organizational requirements. 
The potential to integrate the intervention into an already 
existing health system should also be taken into account [26]. 

The monetary input (e.g. capital investments and operational 
costs) required from the health system to implement and to 
keep providing the intervention [26]. 

Safety is the practical certainty that adverse effects to patients 
or providers will not result from exposure to an intervention 
under defined circumstances [27]. 



The ability of the intervention to be reached by the target 
population, independent of their living place [26]. 



Patients with a different socioeconomic status or a different 
income should be able to make equal use of the intervention [32]. 



10 



10 



Ability to train and deliver all clinical and organizational 8.67 (5-15) 10 

requirements to run the intervention. 

0 poor ability 

1 moderately good ability 

2 good ability 

0 poor affordability (e.g. costs > 1 US$ per capita) 8.47 (0-20) 

1 moderate affordability (e.g. costs > 0.50 < 1 US$ per capita) 

2 good affordability (e.g. costs < 0.50 US$ per capita) [26] 

0 there is a risk of severe adverse effects (life threatening) to 7.87 (0-15) 
patients or a risk of adverse effects (of any kind) to providers 

1 there is a risk of mild adverse effects to patients 

2 there is no risk or a risk of very mild adverse effects (adverse 
effects which will completely recover within a month) to patients 

0 the intervention does not cover (most) people who live far 5.47 (0 - 13) 5 
away from cities. 

1 the intervention does not cover some people who live far away 
from cities. 

2 the intervention covers (almost) all people 

0 the intervention is not accessible to many patients 10.6 (0 -20) 13 

1 the intervention is not accessible to some patients 

2 the intervention is accessible to (almost) all patients 



*weights were calculated by 
NOTE: References were used 



asking participants to divide 100 points over the criteria according to their relative importance for the evaluation of breast cancer interventions. 

to identify the criteria in first instance. The Delphi study may have resulted in adaptations in definitions or scoring scales than originally found in the literature. 
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I 



Formation of 
consultation panel 



Implementation of 
rationing decision 
and evaluation 
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Figure 2 Elements of a priority setting process based on MCDA [16]. 
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The difficulty of avoiding overlap between criteria may 
be explained by a lack of a broader theory on the rela- 
tionship between criteria. Some disagreement between 
participants remained until the end of the process, and 
some overlap was still suspected in the final criteria list. 
These potential overlaps will need attention in the fur- 
ther development of this tool because criteria should 
preferentially be independent from each other [15,42]. 
Especially effectiveness has a risk of overlap with other 
criteria, like cost-effectiveness, safety and geographical 
coverage. Further overlap between criteria should be 
identified and distinctions should be made and clearly 
described in the definitions. 

Limitations of the tool 

The tool also has some practical limitations that one 
should be aware of. First, the tool does not provide guid- 
ance to convert the performance matrix into a final 
prioritization of interventions. This tool stops at rating 
interventions after which a choice should be made based 
on a democratic learning process (Figure 1). This tool 
does not facilitate a democratic learning process, which 
makes it less likely that good rated interventions are im- 
plemented. The accountability for reasonableness frame- 
work (A4R) is successful in introducing such a learning 
process [43]. We recommend making the tool part of 
the integrated MCDA A4R approach for priority setting 
in health as proposed by Baltussen et al. [16], however 
local capacity should be present or established to facili- 
tate such a complete process. 

Second, the proposed rating tool is based on decision- 
maker values and preferences while the views of other 



stakeholder groups are also considered important in pri- 
ority setting exercises. Different stakeholder groups are 
likely to have different preferences for criteria [22,44]. 
This limitation of the tool could be solved while apply- 
ing the tool in a local setting. At that stage, other stake- 
holder groups (like patients, the public, and health care 
workers) can be asked to comment on the relevance of 
the criteria included in the tool and the relative import- 
ance and the tool can be adapted accordingly. 

Third, there are limitations to the collection of infor- 
mation, and it may sometimes be difficult to assess inter- 
ventions on certain criteria. This is however a problem 
across the field of health priority setting and we recom- 
mend to be transparent on the available evidence and its 
quality. A sensitivity analysis may help to give insight in 
the uncertainty of the scoring performances of interven- 
tion options. In this way, quality of evidence is not used 
as a single criterion but as an uncertainty factor per cri- 
terion per intervention [45]. 

Conclusion 

This study describes the development of a rating tool to 
assess the impact of breast cancer interventions on mul- 
tiple criteria. This tool may be a starting point for local de- 
cision makers that would like to conduct multi criteria 
decision analysis to set priorities for breast cancer control. 

Additional file 



Additional file 1: Key considerations for the development of the 
rating tool. In this additional file we elucidate on the key considerations 
made for development of the rating tool. 
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